Robust Multi-Branch Tomlinson-Harashima Precoding in Cooperative MIMO Relay Systems

This paper proposes the design of robust transceivers with Tomlinson-Harashima precoding (THP) for multipleinput multiple-output (MIMO) relay systems with amplify-andforward (AF) protocols based on a multi-branch (MB) strategy. The MB strategy employs successive interference cancellation (SIC) on several parallel branches which are equipped with different ordering patterns so that each branch produces transmit signals by exploiting a certain ordering pattern. For each parallel branch, the proposed robust nonlinear transceiver design consists of THP at the source along with a linear precoder at the relay and a linear minimum-mean-squared-error (MMSE) receiver at the destination. By taking the channel uncertainties into account, the source and relay precoders are jointly optimised to minimise the mean-squared-error (MSE). We then employ a diagonalization method along with some attributes of matrix-monotone functions to convert the optimization problem with matrix variables into an optimization problem with scalar variables. We resort to an iterative method to obtain the solution for the relay and the source precoders via Karush-Kuhn-Tucker (KKT) conditions. An appropriate selection rule is developed to choose the nonlinear transceiver corresponding to the best branch for data transmission. Simulation results demonstrate that the proposed MB-THP scheme is capable of alleviating the effects of channel state information (CSI) errors and improving the robustness of the system.